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Abstract 



Indirect reciprocity is a reputation-based mechanism for cooperation in social dilemma situa- 
tions when individuals do not repeatedly meet. The conditions under which cooperation based 
on indirect reciprocity occurs have been examined in great details. Most previous theoretical 
analysis assumed for mathematical tractability that an individual possesses a binary reputation 
value, i.e., good or bad, which depends on their past actions and other factors. However, in 
real situations, reputations of individuals may be multiple valued. Another puzzling discrep- 
ancy between the theory and experiments is the status of the so-called image scoring, in which 
cooperation and defection are judged to be good and bad, respectively, independent of other 
factors. Such an assessment rule is found in behavioral experiments, whereas it is known to be 
unstable in theory. In the present study, we fill both gaps by analyzing a trinary reputation 
model. By an exhaustive search, we identify all the cooperative and stable equilibria composed 
of a homogeneous population or a heterogeneous population containing two types of players. 
Some results derived for the trinary reputation model are direct extensions of those for the 
binary model. However, we find that the trinary model allows cooperation under image scoring 
under some mild conditions. 



1 Introduction 



Humans and other animals often cooperate even when cooperation is more costly than defec- 
tion. In such social dilemma situations, direct reciprocity is among main reasons for cooperation 



between pairs of individuals that repeatedly meet each other ( Trivers, 1971 Axelrod, 1984 ). 
However, individuals, in particular humans, cooperate with others even when they seldom meet 
the same partners more than once, as is the case for large populations. Reputation-based in- 
direct reciprocity (also called downstream reciprocity; we simply call it indirect reciprocity in 
this paper) seems to be a dominant mechanism that enables cooperation in this situation. In 
indirect reciprocity, individuals cooperate with others with good reputations and they in turn 
gain good reputations if they appositely behave (e.g., cooperate) toward somebody else. The 
conditions under which indirect reciprocity realizes cooperation have been theoretically and 
numerically clarified in great details QNowak fc Sigmund, 1998a] |Nowak fc Sigmund, 1998b" 
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Ohtsuki, 2004 
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Roberts, 2008 Uchida, 2010 Nakamura fc Masuda, 2011 Berger, 2011 Sigmnud, 2012). 



Under the so-called image scoring, cooperation and defection are regarded to be good and 
bad behavior, respectively ( Nowak fc Sigmund, 1998~a Nowak fc Sigmund, 1998b ). Laboratory 
experiments suggest that humans use image scoring to evaluate others' behavior QWedekind fc Milinski, 20 



Milinski et al, 2001 Seinen fc Schram, 2006). However, main theories attain that image scor- 



ing does not stabilize cooperation (Leimar fc Hammerstein, 2001 Panchanathan fc Boyd, 2003 



Ohtsuki, 20041 IQhtsuki fe Iwasa, 2004] |Ohtsuki fe Iwasa, 200"7l |Roberts, 2008|). Although some 



studies have shown the viability of cooperation under image scoring (|Nowak fe Sigmund, 1998a 



Fishman, 20031 |Brandt fc Sigmund, "20041 |Brandt fc Sigmund, 20051 [Brandt fc Sigmund, 20061 



Uchida, 2010), the situations in which cooperation occurs are, in our view, quite restricted (but 



see ( Berger, 2011 ); we discuss this reference in Discussion). Under image scoring, cooperation 
occurs only when individuals always cooperate in the first round (Nowak & Sigmund, 1998a), 
unconditional defectors sometimes cooperate (Fishman, 2003), the number of interaction obeys 
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the binomial distribution ( Brandt fc Sigmund, 2004 ) or Poisson distribution ( Brandt fc Sigmund, 2006 ), 
the probability that individuals recognize others' reputations increases in time (|Brandt fc Sigmund, 2005]) , 



or the reputations of individuals are revealed to others with a small probability ( Uchida, 2010 ). 
Therefore, the reason for the discrepancy between the experiments and theory remains obscure. 
For mathematical tractability and possible influences of the first seminal theoretical pa- 



pers on this subject flNowak fc Sigmund, 1998a] |Nowak fc Sigmund, 1998b[ ), most theoretical 
results of indirect reciprocity are derived from the analysis of binary reputation models. In 
other words, individuals are endowed with the binary reputation, i.e., good (+) or bad (— ), de- 
pending on the last action toward others and other factors. However, the binary reputation may 
not be realistic in that only the last behavior of an individual in the social dilemma situation 
determines the reputation of the individual. In fact, experimental (Wedekind fc Milinski, 2000 



Milinski et a/., 20011 ISeinen fc Schram, 20061 IMilinski et aL 20021 IWedekind fc Braithwaite, 2002 



Keser, 20031 |Bolton fc Katok, 2004[ |Bolton et aL, 20051 |Engelmann fc Fischbacher, 2009[) and 



numerical ( Diekmann fc Przepiorka, 2005 ) studies of indirect reciprocity in the context of on- 
line marketplaces often assume that the reputations are many valued, which complies with the 
reality of online marketplaces ( Resnick fc Zeckhauser, 2002 Resnick et aL, 2006 ). More than 
binary valued reputations have also been employed in numerical studies of indirect reciprocity in 
theoretical biology literature (Nowak fc Sigmund, 1998b Leimar fc Hammerstein, 2001 Mohtashemi fc Is 



Roberts, 2008). Nevertheless, these studies are not concerned with relationships between the 



degree of cooperation and the number of the possible reputation values. 

In this paper we analyze a trinary reputation model to identify stable populations that 
realize cooperation. The difference between the present results and those derived from the 
binary reputation models is remarkable. In particular, we find that image scoring can stabilize 
cooperation in the trinary reputation model. 
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2 Model 



2.1 Donation Game with Reputations 



We consider an infinitely large population. In each generation, the so-called donation game is 
repeated for sufficiently many rounds. Figure [U(A) illustrates the interaction in each round. 
Two players are randomly selected from the population, one as donor and the other as re- 
cipient, with the equal probability. The donor intends to cooperate (C) or defect (D) toward 
the recipient according to the action rule a, which we define below. We assume that the 
donor misimplements intended C such that the donor actually defects with probability e; > 
and that the intended D is always correctly implemented. We seek the possibility of coop- 
eration in the population under this kind of implementation error, which is adverse to coop- 
eration dPanchanathan fc Boyd, 2003[ |Fishman, 20031 |Ohtsuki, 20041 |Ohtsuki fc Iwasa, 2004 



Ohtsuki & Iwasa, 2006 


; Ohtsuki & Iwasa, 2007; 


Nowak & Sigmund, 2005 


Brandt & Sigmund, 2004 



Brandt fc Sigmund, 2005] |Brandt fc Sigmund, 20061 |Uchida, 2010| |Berger, 2011|). If the donor 



implements C, the donor pays cost c, and the recipient obtains benefit b. If the donor imple- 
ments D, the payoffs to the donor and recipient do not change. We assume that < c < b such 
that the donation game is essentially the prisoner's dilemma. 

We assume that each player possesses a reputation that takes one of the three values, i.e., 
G (Good), N (Neutral), or B (Bad). The action rule a is a function from the recipient's 
reputation to the donor's intended action (i.e., C or D). Therefore, there are 2 3 = 8 action 
rules, as shown in Fig. QJB). For example, the A11C and A11D intend C and D regardless of 
the recipient's reputation, respectively. The so-called generous discriminator (gDisc) intends 
C when the recipient's reputation is either G or N and D otherwise. The so-called rigorous 
discriminator (rDisc) intends C when the recipient's reputation is G and D otherwise. 

At the end of each round, the observer assesses the donor to be + or — depending on 
the donor's implemented action (i.e., C or D) and the recipient's reputation (i.e., G, N, or 
B). Such an assessment rule is a function from the donor's implemented action and the re- 
cipient's reputation to the observer's assessment and is called the second-order social norm, 
which extends the concept in the case of the binary reputation (Leimar fc Hammerstein, 2001 
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Nowak fc Sigmund, 2005 ). There are 2 2x3 = 64 social norms. For example, under the social 
norm called image scoring (Fig. QJC)), the observer assigns + and — to the donor's actions 
C and D, respectively, regardless of the recipient's reputation. Another social norm named 
scoring-standing is also shown in Fig. Q^C). Finally, we assume that the assessment (i.e., + 
or — ) that the donor receives is opposite to the one assigned by the observer with probabil- 



ity e a > (Qhtsuki fc Iwasa, 2004 Ohtsuki fc Iwasa, 2007; Nakamura fc Masuda, 2011). We 



assume that the donor's reputation is publicly shared in the population. In other words, the 
donor's new reputation is instantaneously known to all the players in the population. 

We repeat many rounds of the game and define the payoff to each player as the sum of the 
payoff obtained in the games that involve the player. Because the population is infinite, each 
pair of players plays the game at most once such that direct reciprocity is excluded. 

For a later use, we mention four representative second-order social norms in the binary rep- 
utation model (Fig. H^D)). Image scoring ("scoring" in Fig. DJD)) does not stabilize cooperation 



( Panchanathan & Boyd, 2003 


Ohtsuki, 2004 


Ohtsuki fc Iwasa, 2004 


Ohtsuki fc Iwasa, 2007 




Roberts, 2008 


) unless somewhat strong conditions are met ( 


Nowak & Sigmund, 1998a 


Fishman, 2003 


Brandt & Sigmund, 2004; Brandt & Sigmund, 2005 


Brandt & Sigmund, 2006 Uchida, 2010) 



(but see ( |Berger, 2011[ )). Simple standing ("standing" in Fig. CE^D)) and stern judging ("judg- 



ing" in Fig. QJD); also called Kandori ( |Kandori, 1992[ )) are known to stabilize cooperation 
(Ohtsuki fc Iwasa, 2004). Shunning stabilizes cooperation under certain conditions (Ohtsuki fc Iwasa, 20( 



Nakamura fc Masuda, 2011[). 



2.2 Reputation Dynamics 

After each round, the donor's reputation is updated on the basis of the observer's assessment. 
Basically, the donor's reputation shifts upward and downward if the donor receives + and 
— , respectively. When the reputation is binary (i.e., G and B), G and B are equivalent to 
+ and — that the donor receives in the last game, respectively (Nowak fc Sigmund, 1998a 



Nowak fc Sigmund, 2005 ). However, the relationship between the reputation and assessment 
is not straightforward in the trinary reputation model. We assume that reputation dynamics 
obey a Markov chain. We consider the reputation dynamics illustrated in Fig. [2j 
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In the reputation dynamics shown in Fig. |2JA), which we call the gradual dynamics, the 
reputation is assumed to move by at most one level in each round. The reputation is unchanged 
with probability a (0 < a < 1). 

In the reputation dynamics shown in Fig. EJ^B), which we call the saltatory dynamics, the 
donor's reputation can transit from G to B or vice versa in one step. When a G donor receives 
— , the donor's new reputation becomes B and N with probabilities (3d and 1 — /3d, respectively 
(0 < /3d < 1). When a B donor receives +, the donor's new reputation becomes G and N with 
probabilities (3 U and 1 — f3 u , respectively (0 < f3 u < 1). When (/? d , /3 U ) = (1,0), the reputation 



dynamics are similar to the so-called T-period punishment with T = 2 ( Kandori, 1992 ) in 
which players have either state (innocent), 1 (guilty and no repent), or T (guilty and 
T — 1 times of repentant behavior). Guilty players in state 1 regain the innocent state by 
cooperating with innocent players successive T times. Therefore, the states 0, 1, and 2 are 
similar to reputations G, B, and N, respectively, in our model. When (^ d , /3 U ) = (0,1), the 
reputation dynamics are similar to the so-called tolerant scoring ( |Berger, 20TT] ) because the 
donor's reputation becomes B if and only if the donor receives — in the last two rounds, and 
the donor obtains a G reputation if the donor cooperates just once. 



3 Analysis Methods 

3.1 Homogeneous Populations 

We first examine the stability of a homogeneous population of resident players with action 
rule a against mutants with different action rules under a given social norm. Let pq, p^, and 
Pb be the probabilities that the reputation of a resident player is G, N, and B, respectively. 
After a transient of the reputation dynamics, the three probabilities converge to the equilibrium 
values denoted by p* Gl p^, and p^. For expository purposes, we focus on the gradual reputation 
dynamics (Fig. E](A)) in this section. The following calculations are similar for the saltatory 
reputation dynamics (Fig. E](B)); the corresponding results are shown in Appendix A. For the 
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gradual reputation dynamics, we obtain 

P*g = P* G [<x+(l-a)$*]+I%(l-a)Q*, 

p* N = p G (l-a)(l-$*)+p* N a+p B (l -a)$*, (1) 
p B = p* N (l - a)(l -$*) + P B [a + (l-a)(l -$*)], 

where $* is the probability that the donor receives + in the equilibrium. Equation ([T]) and the 
normalization p* G + p^ + p B = 1 lead to 

/ ^,*2 (l — $*) 2 \ 

(Pg,Pn,Pb) = yi_$* + $*2> 1 _$* + $ *3' x_$* + $*2 J • ( 2 ) 

It should be noted that Eq. ([2]) and the following results are independent of the value of a. 
Equation ([2]) implies that the distribution of the reputation is monotonous in the sense that 

P*g > Pn > Pb if $ * > °- 5 an d Pg < Pn < Pb if $ * < 
$* is given by 

= E P^Cr^r + a-Cr)^], (3) 
re{G,N,B} 

where £ r represents the probability that the donor's implemented action is C when the recipient 
has reputation r £ {G,N,B}. ( r has a one-to-one correspondence with action rule a. For 
example, A11C, gDisc, rDisc, and A11D are equivalent to (Cg>Cn,Cb) = (1 — e i? 1 — e ; , 1 — ej, 
(1 — €{, 1 — e i7 0), (1 — 6i, 0, 0), and (0,0,0), respectively. $c,r and $d,t- in Eq. (J3]) are the 
probabilities that the donor receives + when the donor's action is C and D, respectively, and 
the recipient has reputation r £ {G,N,B}. For example, under image scoring (Fig. QJC)), 
$c,r = 1 — e a and $D,r = e a for any r. Under the so-called scoring-standing (see Sec. 14.11 for 
the notation of the social norms) shown in Fig. QJC), $c, r = 1 — ^a for any r, $d,+ = ^d,o = e a> 
and $d,- = 1 ~ ^a- Each term on the right-hand side of Eq. ([3]) is a multiplication of three 
probabilities, i.e., (i) probability p* that a recipient with reputation r is selected, (ii) probability 
( r or 1 — £ r that a donor implements C or D, respectively, and (iii) probability $c,r o r <&n,r 
that the observer assigns + to the donor. 

We substitute p G , p^, and p B obtained from Eq. fl2]) in the right-hand side of Eq. ([3]), which 
we denote by /($*). We obtain $* by solving x = f(x), < x < 1. Because /(x) has a 
quadratic numerator and denominator in terms of x, equation x = f(x) has at most three 
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solutions. Under any pair of action rule and social norm, to which we refer as action-norm pair 
in the following, < e a < f(x) < (1 — ei) (1 — e a ) < 1 holds true for any < x < 1. Therefore, 
the iteration scheme, in which we start with an initial x value (0 < x < 1) and repeatedly apply 
/, always converges. If the iteration starting from x = and that starting from x = 1 converge 
to the same value, the solution specified by $* and {Pg>Pn'Pb} * s unique. We confirmed that 
x = f(x) has a unique solution for each of the 8 x 64 = 512 action-norm pairs. 

Let \I/((T, {pg,Pn,Pb}) be the probability that a donor with action rule a cooperates with a 
recipient randomly chosen from a population according to reputation distribution {pg,Pn,Pb}- 
We obtain 

*0, {pg,Pn,Pb}) = PgCg +PnCn +PbCb- (4) 
The average payoff per round to a resident player in the homogeneous population is given by 

7T = _ C tt(<7, {p* G , P * N ,p* B }) + 6tt(<7, {p* G ,P* N ,P B }). (5) 

To examine the stability of the homogeneous population, we consider an infinitesimally small 
fraction of mutants with action rule o'{^ a) that invades the homogeneous resident population. 
The equilibrium probability distribution of the mutant's reputation, denoted by {p'g,PntPb}i 
the equilibrium probability that a mutant receives +, denoted by $'*, and payoff to a mutant 
player, denoted by ir', are given by 

/ fo>*2 - (I — (J)'*") 2 \ 

(Pg,p£,Pb*)= 2. 2 > 2 ) . ( 6 ) 

** = E P;[C*a,r + (l-C)*D,r], (7) 

rG{G,N,B} 

and 

vr' = -c*(a , ,{p^^,^}) + 6W(t7,{p3,p{J,^}), (8) 

respectively. In Eq. (J^J), C represents the probability that a mutant cooperates with a recipient 
with reputation r. It should be noted that p* in Eq. (|7j) is the solution of Eqs. ([2]) and and 
that $c,r and $D,r also refer to the values for the resident population. 

Action rule a adopted by the resident players is strict Nash equilibrium if the payoff to a 
resident player (i.e., ir) is larger than the payoff to any mutant player (i.e., ir'). Using Eqs. (J5J) 



and (jSJ), we obtain this condition as follows: 

-Ma,{p^p^p^})-^(a,{p* G ,p* N ,p* B })] 

(9) 

<^(V, {pg>pn>pb}) - {pg>pn>pb}) for an y °"' ^ °"- 

We also investigate the stability of action rule a against invasion by a previously iden- 
tified strong competitor (Leimar Hammerstein, 2001), which is so-called the Self strategy 



(Ohtsuki & Iwasa, 2004). A Self donor plays a donation game as selfishly as possible under the 
constraint that the donor's reputation stays above a threshold. When determining the action, 
the Self donor refers to its own reputation and does not refer to the recipient's reputation. We 
assume that the Self donor defects if its reputation is G and cooperates otherwise. Under the 
gradual reputation dynamics, the Self player maintains its reputation value at G or N, not B, 
except in the case of error. 

3.2 Heterogeneous Populations Composed of Two Action Rules 

We also examine the stability of heterogeneous populations in which two action rules, denoted 
by o\ and 02, coexist with fractions q\ and 02, respectively (qi + #2 = 1). For each of (qi, 92) = 
(0.01,0.99), (0.02,0.98), (0.99,0.01)), we first calculate the equilibrium probabilities of + 



for <Ti and o"2 by an iteration scheme similar to that described in Sec. 13.11 Under the gradual 
reputation dynamics, the distribution of the reputation values is given by 

/ <|>* 2 _ $*) (1 _ $*)2 \ 

(p*G,i,ph>pki) = [ 1 _ $ ; + $ * 2 » 1 z + $*2 , 1 _ $ , + ' ^2) , ( 10 ) 

where p* ri is the equilibrium probability that a player with action rule i (i = 1,2) possesses 
reputation r G {G, N, B}, and $* represents the equilibrium probability that a cr, player receives 
+ . The equivalent of Eq. ffT0|) and the following results can be obtained similarly for the 
saltatory reputation dynamics. $* is given by 

2 

$ i=E E ^[CrA.r + a-Cr.O^D.r], (H) 
j=l rG{G,N,B} 

where Cr,i is the probability that a 0{ donor implements C when the recipient has reputation r. 
Substitution of Eq. ^ into Eq. CP leads to ^ = where ^ = 
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It should be noted that $* = g($>*) may have multiple fixed points. An example is given by 
the population composed of the equal fraction of o\ = gDisc and a 2 = rDisc, i.e., qi = q 2 = 0.5, 
under image scoring. In this case, both a cooperative population (i.e., $*, $2 ~ (1 — e i)(l — e a)) 
and a defective population (i.e., $2 ~ e a) satisfy $* = g (<&*). Because of the multistability, 
we adopt ll 2 = 121 initial conditions, i.e., $ = (0.1z,0.1j), < i,j < 10, for the iteration 
scheme to identify all the fixed points. In fact, we find that the multistability does not cause a 
severe problem. We will show in Results that five mixed populations composed of two action 
rules are stable and realize a sufficiently large probability of cooperation. Among them, only 
one population, which consists of gDisc and rDisc and is stable under the so-called scoring- 
shunning social norm, yields the bistable equilibria. One equilibrium yields a large cooperation 
probability (> 0.93) and the other equilibrium yields a low cooperation probability (< 0.03). 
These results are qualitatively the same for the gradual and saltatory reputation dynamics. For 
this social norm, we only keep the more cooperative equilibrium. 

Then, we obtain the trinary distribution of reputation for each of the two action rules by 
substituting <3>* in Eq. (TIT]) . The average payoff per round to a <jj resident player (i — 1,2) is 
given by 

2 2 

j=i 3=1 

where ^(aj, {Pgj,P^j,Pbj}) is the probability that a o"j donor cooperates with a aj recipient. 
In the equilibrium, the payoffs to the players with the different action rules are the same. 
Therefore, we calculate the value of b/c for which tx\ = ir 2 . If d(7Ti — vr2)/dgi > 0, the mixed 
population is unstable against an infinitesimally small drift of the fraction of the two action 
rules. We are concerned with the pairs of <j\ and a 2 that satisfy d(7Ti — 7T2)/dgi < at the 
obtained b/c value. 

The mixed population is strict Nash when the payoff to any of the six mutants with a third 
action rule is smaller than that to a resident player. By substituting Eq. ([6]) in 

2 

$ '*=E E ^j[C*C,r + (l-C)*D.r], (13) 
3=1 re{G,N,B} 

we obtain the equilibrium probability that a mutant receives + (i.e., $'*) and the equilibrium 

9 



distribution of the mutant's reputation (i.e., {p'giPnjPb})- The payoff to a mutant player is 
given by 

2 2 

3=1 J=l 

As in the analysis of the homogeneous population, we also examine the stability of the hetero- 
geneous population against invasion by the Self mutant (Sec. 13. ip . 

We do not examine the mixed population composed of more than two action rules. 



4 Results 

We refer to each action rule by concatenating three letters, either C or D. The first, second, 
and the third letters represent the intended action toward a recipient with reputation G, N, 
and B, respectively. For example, gDisc = CCD and rDisc = CDD. 



4.1 Gradual Reputation Dynamics 
4.1.1 Enumeration of Stable Populations 

We set €i = e a = 0.02 and consider the gradual reputation dynamics (Fig. E£A)). We found 
that the homogeneous population is stable against invasion by mutants for some benefit-to- 
cost values b/c for 108 out of 8 x 64 = 512 action-norm pairs. We exclude 64 pairs with 
a = A11D from the 108 pairs because of the lack of cooperation. The entire game is symmetric 



with respect to the simultaneous flipping of G f)- B and + <H- — ( Ohtsuki fc Iwasa, 2004 ). 
Therefore, there are (108 — 64)/2 = 22 essentially distinct pairs. The 22 pairs are listed in 
Table [TJ In addition, there are nine essentially distinct mixtures of two action rules that are 
stable under a certain social norm. Among these stable populations, there are 12 homogeneous 
populations composed of a single action rule and five heterogeneous populations composed of 
two action rules that realize a probability of cooperation larger than 0.5 for a, b/c value smaller 
than 20. Our additional numerical simulations suggest that the probability of cooperation tends 
to unity if and only if this criterion is satisfied (Appendix B). 
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The cooperative equilibria, i.e., stable populations satisfying this criterion under a given 
social norm, are summarized in Fig. In Fig. E^A), a social norm is represented by a 

combination of sq, sn, and sb, each of which takes either ++, — h, , or H — . For example, 

sq = — h indicates that donor's implemented action C and D toward a G recipient is assessed 
to be — and +, respectively. In fact, all the cooperative equilibria require sq = H — such that 
only sn and sb are indicated in Fig. G9(A). 

4.1.2 s N = + + or - + 

Six action-norm pairs having a = rDisc, sq = H — , = ++ or — h, and Sb = ++, — h, or 

realize a large probability of cooperation (« 0.94), a large probability of + (~ 0.98), and a mild 
restriction on b/c (i.e., 6/c > 1.004). In these equilibria, most resident players have the G 
reputation because the probability of + is large. A small fraction of players possesses the N 
reputation owing to error (see Sec. 12.11 for the definition of two types of error), and an even 
smaller fraction of players possesses the B reputation. We verified that the population is stable 
against invasion by Self players (Sec. 13. 1 p under each of the six social norms. 

We call the six social norms standing-standing, standing-judging, standing-shunning, judging 
standing, judging-judging, and judging-shunning. The first (second) half of the name repre- 
sents the social norm represented by sq and sn {sq and sb) in the case of the binary reputation 
model. For example, the combination of Sq = H — and sn = ++ represents the standing 
social norm in the binary model if N is identified with B in the binary model (Fig. [TJD)). 
Similarly, the combination of Sq = H — and Sb = — h represents the judging social norm in the 
binary model. Therefore, we call the social norm given by sq = H — , sn = ++, and sb = — h 
standing-judging. 

The six norms realize nearly perfect cooperation because the first half of the social norm 
(i.e., combination of sq and sn) is either standing or judging and the rDisc donor cooperates 
with G recipients, but not N recipients. It should be noted that standing and judging are 
the only second-order social norms that stabilize cooperation without special conditions in the 
binary reputation model (Ohtsuki fc Iwasa, 2004). Even if the donor's reputation transits from 



N to B owing to error, the donor regains the N reputation by cooperating with a G recipient, 
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which occupies the majority of the population. Although donors meeting recipients with the B 

reputation receive — under s B = , cooperation at the population level is maintained because 

few players have the B reputation. 

4.1.3 s N = — 

Two cooperative action-norm pairs, i.e., a = rDisc, sq = H — , sn = , and sb = ++ or — h, 

are also stable under a mild condition on b/c, i.e., b/c > 1.018. The homogeneous population 
of rDisc is not invaded by Self strategy under each of the two social norms. However, the 
probability of + (~ 0.74) and that of cooperation 0.66) are not as large as those for the 
previous six action-norm pairs. This is intuitively because, under the current social norms, 
i.e., shunning-standing and shunning-judging, a donor whose recipient has an N reputation 
always receives — except in the case of the assessment error. This behavior of the model is 
similar to that under shunning in the binary reputation model. Nevertheless, different from 
the case of the binary model, the cooperation probability tends to unity in the error-free limit 
(Appendix B) for the two norms in the trinary reputation model. We discuss this point further 
in Discussion. 

4.1.4 s N = H — 

The other cooperative equilibria are four homogeneous populations and five heterogeneous 
populations, which are stable under social norms satisfying s G = s N = H — (Fig. E(A)). For 
sb = ++ (scoring-standing) and — h (scoring-judging), a homogeneous population composed 
of gDisc is stable for large b/c (> 8.480), and a mixed population composed of gDisc and rDisc 

is stable for small b/c (< 8.480). For sb = (scoring-shunning), a homogeneous population 

composed of gDisc is stable for large b/c (> 8.108), and a mixed population composed of gDisc 
and rDisc is stable for small b/c (< 8.108). The fraction of gDisc increases with b/c under 
scoring-standing and is unity when b/c > 8.480, as shown in Fig. [3]^B). The results shown in 
Fig. E](B) are indistinguishable from those for scoring-judging and similar to those for scoring- 
shunning. For the three social norms, the fraction of gDisc converges to 1 — c/b in the limit 
Ci,e a — > (Appendix C), which implies that only the mixed population of gDisc and rDisc 
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is stable in the error-free limit. Consistent with this, the threshold value of b/c above which 
the homogeneous gDisc population is stable diverges as the error probabilities become small 
(Appendix B). 

If we hypothetically merge G and N, our result that cooperation is stable under scoring- 
standing and scoring-judging corresponds to the fact that the cooperation is stable under stand- 



ing and judging, respectively, in the binary model ( Ohtsuki fc Iwasa, 2004 ). Under scoring- 
shunning, cooperation is not undermined by sb = for the same reason as that for standing- 
shunning and judging-shunning (see Sec. 14.1.21 and Discussion). 

Under scoring-scoring (i.e., sq = sn = sb = H — ), which we also call the image scoring 
(Fig. QJC)), there are three types of stable populations depending on the b/c value. When 
1.941 < b/c < 8.230, the mixed population composed of gDisc and CDC is stable (Fig. EJ^C)). 
We regard CDC as a variant of rDisc because there are few players having the B reputation in 
the stable population; a CDC player obtains a slightly larger payoff than an rDisc player. In 
fact, the mixed population of gDisc and rDisc is stable against invasion by all but CDC mu- 
tants. When 8.230 < b/c < 12.53, the homogeneous population of gDisc is stable (Fig. EJ^C)). 
When b/c > 12.53, the mixed population composed of gDisc and A11C is stable (Fig. E(C)). 
In all the three cases, cooperation occurs with a large probability (> 0.94). This result is in a 
stark contrast with that in the binary reputation model, whereby image scoring does not usu- 
ally support cooperation QPanchanathan fc Boyd, 2003| |Ohtsuki, 2004| |Ohtsuki fc Iwasa, 2004 



Ohtsuki fc Iwasa, 2007| |Roberts, 2008[). We discuss this discrepancy in Discussion. 



Under image scoring, the values of b/c and the error probabilities under the assumption 
€i = e a , for which one of the three populations is stable are shown in Fig. IHA). For small 
error, the homogeneous population of gDisc and the mixed population composed of gDisc and 
A11C are invaded by rDisc mutants. Only the heterogeneous population composed of gDisc and 
CDC survives the error-free limit. In this limit, the fraction of gDisc is given by 1 — c/b for 
the same reason as that for the heterogeneous population composed of gDisc and rDisc under 
scoring-standing (Sec. I4.L4"} also see Appendix C). 

The nine populations stable under sq = sn = H — are unstable against invasion by Self mu- 
tants. When there are more than two possible reputation values, Self is generally a strong com- 
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petitor under image scoring such that it undermines cooperation ( Leimar fc Hammerstein, 2001 ). 
Our results are consistent with theirs. In the next section, we propose a different reputation 
dynamics that makes cooperation stable against invasion by Self under image scoring. 



4.2 Saltatory Reputation Dynamics 
4.2.1 Downward Saltation 

We first investigate a saltatory reputation dynamics (Fig. E^B)) given by /?d = 0.5 and f3 u = 0. 
This dynamics implies that a G reputation of a donor jumps down to a B reputation in one 
step with a probability (3 d = 0.5. We set e; = e a = 0.02 and identify stable populations under 
each social norm. To select cooperative equilibria, we imposed the same condition as that in 
the case of the gradual reputation dynamics, i.e., a cooperation probability larger than 0.5 for 
some b/c < 20. 

We find that the set of cooperative equilibria (without consideration of Self mutants) is the 
same as that for the gradual reputation dynamics (Fig. EJ^A)). The cooperation probability is 
equal to 0.9237 for rDisc under standing-standing, 0.8614 for rDisc under standing-shunning, 
0.6804 for rDisc under shunning-standing, 0.9340 for gDisc under scoring-standing, and 0.9340 
for gDisc under scoring-shunning or image scoring. The cooperation probability is preserved if 
we replace standing by judging. Because a G reputation can turn into a B reputation in one step, 
the probability of cooperation is somewhat smaller than in the case of the gradual reputation 
dynamics. The difference in the cooperation probability between the two reputation dynamics 

is relatively large when sb is or H — . This is because players with the B reputation, albeit 

occupying a small fraction, can make other players with the G reputation to transit to the B 
reputation in one step under these social norms. 

Under image scoring, the stable population as a function of b/c and the error probabilities 
under the assumption e; = e a is shown in Fig. HJ^B). Figure ID^B) is similar to the results for 
the gradual reputation dynamics shown in Fig. IHA). As a slight difference between Figs. ID^A) 
andH^B), less cooperative populations (e.g., a homogeneous population of gDisc as compared 
to a heterogeneous population of gDisc and A11C) can be stable in the downward saltatory 
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reputation dynamics than in the gradual reputation dynamics for the same values of b/c and 
the error probability. This is intuitively because the downward saltatory reputation dynamics 
yields worse reputations than the gradual reputation dynamics. 

We find that Self mutants cannot invade any of the stable populations shown in Fig. [3]^ A). 

4.2.2 Upward Saltation 

We next identified the stable populations under the saltatory reputation dynamics (Fig. 121(B)) 
given by (3d = 0, and /3 U = 0.5. This dynamics implies that a B reputation of a donor jumps up 
to a G reputation in one step with a probability f3 u = 0.5. We set e; = e a = 0.02 and identify 
stable populations under each social norm. To select cooperative equilibria, we imposed the 
same condition as that in the case of the gradual reputation dynamics, i.e., a cooperation 
probability larger than 0.5 for some b/c < 20. 

We find that the set of cooperative equilibria (without consideration of Self mutants) is 
the same as that for the gradual reputation dynamics. The cooperation probabilities in stable 
and cooperative populations are equal to 0.9416 for rDisc under standing-standing, 0.9401 for 
rDisc under standing-shunning, 0.7605 for rDisc under shunning-standing, 0.9785 for gDisc 
under scoring-standing, and 0.9783 for gDisc under scoring-shunning or image scoring. The 
cooperation probability is preserved if we replace standing by judging. Because a B reputation 
can turn into a G reputation in one step, the probability of cooperation is a little larger than in 
the case of the gradual reputation dynamics. The difference is relatively large when sn = • 

Under image scoring, the stable population as a function of b/c and e; (= e a ) is shown in 
Fig. HJC). Figure HJC) is quantitatively different from the results for the gradual reputation 
dynamics shown in Fig. IHA). More cooperative action rules (e.g., a heterogeneous population 
of gDisc and A11C compared to a homogeneous population of gDisc) are stable in the upward 
saltatory reputation dynamics than in the gradual reputation dynamics for the same values 
of b/c and the error probability. This is intuitively because the upward saltatory reputation 
dynamics yields better reputations than the gradual reputation dynamics. 

We find that Self mutants cannot invade a homogeneous population of rDisc under each of 
the eight social norms satisfying sq = H — and sn = ++, — h, • However, under social 
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norms satisfying sq = sn = H — , with image scoring included, the nine populations are not 
stable against invasion by Self mutants. The result that Self undermines cooperation is the 
same as that for the gradual reputation dynamics (Fig. HJ^A)). 



5 Discussion 

We analyzed a trinary reputation model of indirect reciprocity and identified cooperative Nash 
equilibria composed of a single action rule or mixture of two action rules. Independent of details 
of the reputation dynamics (i.e., gradual or saltatory), we found at a small error level (i.e., 
c; = e a = 0.02) that 12 homogeneous populations and five mixed populations are cooperative 
and stable under different social norms (Fig. HHA)). When the error probabilities are even 
smaller, eight homogeneous populations and four mixed populations remain stable (Appendix 
B). In particular, under image scoring, the heterogeneous population composed of gDisc and 
CDC (a variant of rDisc) is stable in the error-free limit (Fig. 11(A); see Fig. [1(B) for the 
definition of gDisc, CDC, and rDisc). 

The results derived from the trinary reputation model and those derived from the binary 
reputation model (Nowak fe Sigmund, 1998a) are similar in some aspects. For example, the 



standing-standing social norm (i.e., Sq = H — , Sn = Sb = ++) m the trinary model coincides 
with the standing social norm in the binary model (Fig. [1(D)) if we merge the N and B repu- 
tations in the trinary model. Therefore, the result that standing-standing enables cooperation 
is not surprising. Similarly, scoring-judging (i.e., Sq = Sn = H — , sb = — h), for example, is 
almost equivalent to judging in the binary model if we merge G and N reputations in the trinary 
model. However, the results for the trinary and binary models are fundamentally different in 
the following two aspects. 

First, shunning (see Fig. QJD) in the case of the binary reputation model) is more supportive 
of cooperation in the trinary than binary model. In the binary model, shunning results in 
a cooperation probability of ~ 1/2 in the error-free limit unless a cooperation prone initial 
condition and a finite number of rounds are combined (Qhtsuki fc Iwasa, 2007) or reputations 



of players are only partially visible to others (Nakamura fc Masuda, 2011). In the trinary 
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model, shunning-standing and shunning-judging stabilize full cooperation in the error-free 
limit (Appendix B). Under these social norms, a donor meeting a recipient with an N reputation 
receives — irrespective of the action, which is common to the behavior of the binary model. In 
the trinary model, however, even if a donor obtains a B reputation, the donor easily receives + 
either by cooperating with a G recipient or justifiably defecting against a B recipient. Owing 
to the contribution of the justified defection, a donor gains + more often than — on average. If 
players with the G reputation increase in number owing to this mechanism, players more likely 
receive + than — . This positive feedback sustains cooperation under shunning-related social 
norms in the trinary model. 

Second, and the more important, image scoring is capable of supporting cooperation in our 
model. Even if we consider Self mutants, which avoid a B reputation and are as selfish as 
possible, image scoring supports cooperation if the probability that a G reputation transits 
to a B reputation in one step is positive. In previous literature, the Self strategy is recog- 
nized as a strong competitor that spoils cooperation in the indirect reciprocity game with more 
than two possible reputation values ( Leimar fc Hammerstein, 2001 ). Our conclusion that image 
scoring can support cooperation is consistent with the results derived from behavioral exper- 
iments dWedekind fc Milinski, 2000] |Milinski et al, 2001[ |Seinen fc Schram, 2006} and those 
derived from numerical simulations ( Nowak fc Sigmund, 1998b Diekmann fc Przepiorka, 2005 ) 
but opposite to those derived from the binary reputation model QPanchanathan fc Boyd, 2003 



Ohtsuki, 2004| |Ohtsuki fc Iwasa, 2004| |Ohtsuki fc Iwasa, 2007[) . We reached this conclusion by 



simply introducing a third reputation to the standard binary model of indirect reciprocity. The 
cooperation under image scoring in our model does not require forced cooperation in the first 
round ( Nowak fc Sigmund, 1998a ), partial cooperation of defective players ( Fishman, 2003 ), bi- 
nomially or Poisson distributed number of rounds (|Brandt fc Sigmund, 2004{ |Brandt fc Sigmund, 2006[), 



growth of social networks used for transmission of reputation (Brandt fc Sigmund, 2005), or a 



small probability with which the donor's reputation is revealed to other players ( Uchida, 2010 ). 

In our model, cooperation under image scoring occurs for the following intuitive reason. 
Although the composition of the stable population depends on the benefit-to-cost ratio and 
the error probabilities, the main action rule present in the stable population is gDisc, which 
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cooperates with G and N recipients and defects against B recipients. In the equilibrium, most 
gDisc resident players possess the G reputation. If some A11D mutants are present, a B player 
in the population is likely to be A11D and not gDisc. With the trinary reputation, gDisc players 
justifiably defect (i.e., D against a B recipient) but do not selfishly defect (i.e., D against a 
G or N recipient). A donor that has justifiably defected would receive an N reputation but 
not a B reputation because few recipients have the B reputation in the population. Therefore, 
gDisc players would not obtain the B reputation. gDisc players that happen to obtain the N 
reputation, when selected as donor, likely meet a recipient with a G reputation such that they 
regain the G reputation. In contrast, A11D players would obtain the B reputation such that 
they are not helped by others. This contrasts with the case of the binary reputation model, in 
which a discriminating donor that defects against whatever recipient immediately receives the 
worst reputation (i.e., B) and then is defected by others. 

The presence of downward saltation (i.e., transition from the G reputation to B reputation 
in one step) is a key to make a cooperative population stable against invasion by Self mutants. 
To explain this point intuitively, let us consider a homogeneous population of gDisc and assume 
that the implementation and assessment errors are absent. If downward saltation is absent in 
the reputation dynamics, Self mutants flip between G and N reputations by alternatingly coop- 
erating and defecting. By doing so, the Self mutants can elicit cooperation from gDisc residents 
and cooperate with probability ~ 1/2. Because gDisc residents cooperate with probability ~ 1, 
the gDisc population is invaded by Self mutants. However, if downward saltation can occur 
in the reputation dynamics, a nonnegligible fraction of Self mutants possess the B reputation 
because they defect when they have the G reputation. In contrast, gDisc players maintain the 
G reputation because they cooperate even if they have the G reputation. Therefore, in the 
presence of downward saltation, Self mutants cannot invade the population of gDisc residents. 

The mechanism of cooperation under image scoring in our model is similar to that under the 
so-called tolerant scoring proposed by Berger ( Berger, 2011 ). The population of tolerant dis- 
criminator, which defects against a recipient if the recipient has defected in the last two rounds 
and cooperates otherwise, is stable. Berger's results and ours are different in the following 
aspects. First, Berger assumed three action rules, i.e., tolerant discriminator, which is similar 
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to gDisc, A11C, and A11D and investigated the behavior of the model under tolerant scoring. 
We carried out an exhaustive search of the space of the action rule and social norms to find the 
viability of image scoring. Second, in our model, cooperation is stable even if the reputation 
moves only one step in a round, if the Self mutants are not considered. It should be noted that 
Berger did not consider the Self mutants. Third, cooperation is realized by a homogeneous 
population of one type of players in the Berger's model. In our model, cooperation is realized 
by mixture of two types of discriminative players (i.e., gDisc and CDC). 

The present study has following limitations. First, we assumed that all the players in the 
population use the same social norm. This oversimplification excludes a possible situation 



in which different norms compete in a population (e.g., ( Pacheco et al., 2006 Uchida, 2010 )). 
Second, we only analyzed the stability of populations composed of up to two action rules for 
simplicity. Third, we analyzed local stability of the equilibria and disregarded dynamics. Even 
when a cooperative equilibrium is locally stable, it may in fact attract a tiny fraction of initial 
conditions. Fourth, we assumed that the reputation sharing is public. In other words, the 
information about the donor's new reputation immediately spreads from the observer to the 
entire population. In a large population, such immediate spreading is impossible, and one has 
to assume, for example, that only a fraction of players gains the information about a donor in 
one game QNowak fc Sigmund, 1998b] |Brandt fc Sigmund, 2004| |Uchida, 2010P . 



Appendix A: Distribution of the Reputation for the Salta- 
tory Reputation Dynamics 

Under the saltatory reputation dynamics (Fig. EJ^B)), we obtain 

p* N = pfc(l- #0(1 (15) 
p* B = p^ d (l_$*)+|^(l-$*)+p*(l-$*). 
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Equation ffTBT) and the normalization p^ + p^ + pg = 1 lead to 



Pb 



1 



where Z 



Ai)(i-Ax)$*(i 



(1-/3 U )$* 2 + /3 U $* 
(l-/3 d /3 u )$*(l-$*) 
Ai)(l-$*) 2 + /3 d (l-$* 



(16) 



Appendix B: Justification of the Criterion of Equilibrium 
Selection 

We defined cooperative equilibrium as stable population in which the probability of cooper- 
ation is larger than 0.5 for some b/c such that b/c < 20. To justify this criterion, we carry 
out additional numerical simulations with various error probabilities satisfying = e a . The 
probability of cooperation in each stable homogeneous population is shown for various error 
probability values in Fig. [51 Figure [5] shows that, under 13 out of the 22 stable action-norm 
pairs, the probability of cooperation seems to converge to unity in the error- free limit. It should 
be noted that we identified 12, not 13, cooperative homogeneous populations in the main text. 
One of the 13 action-norm pairs is excluded because it is stable only for large b/c values (i.e., 
b/c > 52.63) at e ; = e a = 0.02 (Tabled]). The probability of cooperation is larger than 0.5 
for some b/c for five out of the nine stable heterogeneous populations composed of two action 
rules. For all the five heterogeneous populations, the probability of cooperation is larger than 
0.5 for a b/c value smaller than 20 at e\ = e a = 0.02. 

In the homogeneous populations, under eight out of the 13 social norms for which the prob- 
ability of cooperation seems to converge to unity in the error-free limit, i.e., standing-standing, 
standing-judging, standing-shunning, judging-standing, judging-judging, judging-shunning, 
shunning-standing, and shunning-judging, the range of b/c in which the corresponding ac- 
tion rule is stable tends to b/c > 1 as the error probabilities become small. For example, 
under standing-standing, standing-judging, judging-standing, and judging-judging, the ho- 
mogeneous population of rDisc is stable in the range b/c > 1.00006 when e\ = e a = 0.0025; 
the corresponding range is b/c > 1.004 when e\ = e a = 0.02 (Table [1]). Under the other five 
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social norms, i.e., standing-scoring, scoring-standing, scoring-judging, scoring-shunning, and 
scoring-scoring, stable cooperation requires a large value of b/c when the error probabilities 
are small. For example, under standing-scoring, the homogeneous population of CDC is stable 
in the range b/c > 402.1 when e\ = e a = 0.0025; the corresponding range is b/c > 52.63 when 
e; = e a = 0.02 (Tabled]). Under image scoring, the homogeneous population of gDisc is stable 
in the range b/c> 66.51 when e\ = e a = 0.0025; the corresponding range is 8.230 < b/c < 12.53 
when e; = e a = 0.02 (Table HJ Fig. g(A)). 

In the heterogeneous populations, four out of five equilibria, i.e., mixture of gDisc and rDisc 
under scoring-standing, mixture of gDisc and rDisc under scoring-judging, mixture of gDisc 
and rDisc under scoring-shunning, and mixture of gDisc and CDC under image scoring, realize 
a large probability of cooperation in a wide range of b/c when e; = e a = 0.02 (Figs. [3](B) and 
E£C)) and also do so when the error probabilities are smaller. For example, the heterogeneous 
population composed of gDisc and CDC is stable under image scoring in the range 1.994 < 
b/c < 66.51 when e\ = e a = 0.0025; the corresponding range is 1.941 < b/c < 8.230 when 
e i = e a = 0.02 (Fig. HA)). 

Appendix C: Fraction of Two Action Rules in the Error- 
free Limit 

We consider a social norm given by sq = = H — and sb = ++, — h, or . In the limit 

6i, e a — > 0, the fraction of gDisc, denoted by g g Disc, and that of rDisc converge to 1 — c/b and 
c/b, respectively, for the following reason. In the equilibrium, few players possess the N or B 
reputation. The behavior of rDisc players and that of gDisc players differ only toward recipients 
with reputation N. An rDisc donor with a G reputation defects against a recipient with an 
N reputation, receives — , and transits to the N reputation. If this rDisc player is selected as 
recipient, an rDisc donor defects and a gDisc donor cooperates. If selected as donor, this rDisc 
player would cooperate because most recipients have the G reputation, receive +, and transit 
to G. Because the rDisc player with an N reputation is selected as recipient once on average 
before selected as donor, the expected payoff to the rDisc donor during this period is equal to 
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<?gDisc& o n average. During the same period, a gDisc donor with a G reputation cooperates with 
a recipient with an N reputation, pays c, receives +, keeps a G reputation, and gains benefit 
b when selected as recipient. By equating the payoffs to rDisc and gDisc players, we obtain 
<7gDiscfr = —c + b, which leads to g g Di SC = 1 — c/b. 
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Figure 1: Rule of the donation game with trinary reputations. (A) Illustration of the inter- 
action in a single game. (B) Eight action rules. (C) Representative social norms. The rows 
represent the donor's actions (i.e., C and D), the columns represent the recipient's reputations 
(G, N, and B), and + and — represent the assessments that observer assigns to the donor. (D) 
Representative social norms in the binary reputation model. 
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Figure 2: Two types of reputation dynamics. (A) Gradual reputation dynamics. (B) Saltatory 
reputation dynamics. $ represents the probability that the donor receives +. 
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Figure 3: Results for the gradual reputation dynamics. We set e x = e a = 0.02. (A) Cooperative 
and stable action rules under different social norms. All the shown social norms own sq = H — . 
(B) Average cooperation probability and the fraction of gDisc players under scoring-standing 
(i.e., sg = sn = H — and sb = ++)• It should be noted that the fraction of gDisc and that 
of rDisc sum to unity. (C) Average cooperation probability and the fraction of gDisc players 
under scoring-scoring (also called image scoring; sq = Sn = Sb = H — )• The fraction of gDisc 
and that of CDC or AIIC sum to unity 
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Figure 4: Parameter regions in which one of the three populations is stable under image 
scoring. We set the error probability e\ = e a . (A) Gradual reputation dynamics, i.e., (f3 d , (3 U ) = 
(0, 0). (B) Saltatory reputation dynamics with downward saltation, i.e., (/3d, /3 U ) = (0.5, 0). (C) 
Saltatory reputation dynamics with upward saltation, i.e., ((3d, j3 u ) = (0,0.5). The cooperative 
population is stable against invasion by Self mutants in the parameter region above the green 
line in (B). Self invades the cooperative population in all the parameter regions in (A) and (C). 



29 




0.01 0.02 0.03 0.04 0.05 
Error probability 



a 


S G 




S B 




gDisc 


+ - 


+ - 


* + 


+ 


gDisc 


+ - 


+ - 


*- 


X 


CDC 


+ - 


+ + 


+ - 




rDisc 


+ - 






□ 


rDisc 


+ - 






■ 


rDisc 


+ - 






o 


DCD 


*- 


+ - 






DCD 


*- 


+ - 


*- 





rDisc 


+ - 






♦ 



Figure 5: Relationships between the probability of cooperation and the error probability for 
the 22 stable homogeneous populations. We assume the gradual reputation dynamics and set 
6i = e a . An asterisk represents either + or — . 
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Table 1: Stable action-norm pairs. We also show the range of b/c in which action rule a 
is stable, probability of C, and mean + assessment (i.e., $*) under the gradual reputation 
dynamics. An asterisk represents either + or — . We set e\ = e a = 0.02. 
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